The present disclosure relates to a triangulation scanner and an integral camera configured to achieve augmented reality (AR).
One set of coordinate measurement devices belongs to a class of instruments that measure the three-dimensional (3D) coordinates of a point by sending a laser beam to the point. The laser beam may impinge directly on the point or on a retroreflector target in contact with the point. In either case, the instrument determines the coordinates of the point by measuring the distance and the two angles to the target. The distance is measured with a distance measuring device such as an absolute distance meter or an interferometer. The angles are measured with an angle measuring device such as an angular encoder. A gimbaled beam-steering mechanism within the instrument directs the laser beam to the point of interest.
The laser tracker is a particular type of coordinate measuring device that tracks the retroreflector target with one or more laser beams it emits. The laser tracker is thus a “time-of-flight” type of measurement device. Coordinate measuring devices closely related to the laser tracker are the laser scanner and the total station. The laser scanner steps one or more laser beams to points on a surface of an object. It picks up light scattered from the surface and from this light determines the distance and two angles to each point. The total station, which is most often used in surveying applications, may be used to measure the coordinates of diffusely scattering or retroreflective targets. Hereinafter, the term laser tracker is used in a broad sense to include laser scanners and total stations.
Ordinarily the laser tracker sends a laser beam to a retroreflector target. A common type of retroreflector target is the spherically mounted retroreflector (SMR), which comprises a cube-corner retroreflector embedded within a metal sphere. The cube-corner retroreflector comprises three mutually perpendicular mirrors. The vertex, which is the common point of intersection of the three mirrors, is located at the center of the sphere. Because of this placement of the cube corner within the sphere, the perpendicular distance from the vertex to any surface on which the SMR rests remains constant, even as the SMR is rotated. Consequently, the laser tracker can measure the 3D coordinates of the object surface by following the position of an SMR as it is moved over the surface. Stating this another way, the laser tracker needs to measure only three degrees of freedom (one radial distance and two angles) to fully characterize the 3D coordinates of a surface.
One type of laser tracker contains only an interferometer (IFM) without an absolute distance meter (ADM). If an object blocks the path of the laser beam from one of these trackers, the IFM loses its distance reference. The operator must then track the retroreflector to a known location to reset to a reference distance before continuing the measurement. A way around this limitation is to put an ADM in the tracker. The ADM can measure distance in a point-and-shoot manner, as described in more detail below. Some laser trackers contain only an ADM without an interferometer. U.S. Pat. No. 7,352,446 ('446) to Bridges et al., the contents of which are incorporated herein by reference, describes a laser tracker having only an ADM (and no IFM) that is able to accurately scan a moving target. Prior to the '446 patent, absolute distance meters were too slow to accurately find the position of a moving target.
A gimbal mechanism within the laser tracker may be used to direct a laser beam from the tracker to the SMR. Part of the light retroreflected by the SMR enters the laser tracker and passes onto a position detector. A control system within the laser tracker can use the position of the light on the position detector to adjust the rotation angles of the mechanical axes of the laser tracker to keep the laser beam centered on the SMR. In this way, the tracker is able to follow (track) an SMR that is moved over the surface of an object of interest.
Angle measuring devices such as angular encoders are attached to the mechanical axes of the tracker. The one distance measurement and two angle measurements performed by the laser tracker are sufficient to completely specify the three-dimensional location of the SMR at any point on the surface of the object being measured.
Several laser trackers have been disclosed for measuring six, rather than the ordinary three, degrees of freedom. These six degrees of freedom include three translational degrees of freedom and three orientational degrees of freedom. Exemplary six degree-of-freedom (six-DOF or 6DOF) laser tracker systems are described by U.S. Pat. No. 7,800,758 ('758) to Bridges et al., U.S. Pat. No. 8,525,983 ('983) to Bridges et al., and U.S. Pat. No. 8,467,072 ('072) to Cramer et al., the contents of each of which are incorporated herein by reference.
An alternative to a time-of-flight measurement device such as a laser tracker is a scanning system that determines the 3D coordinates of an object surface based on triangulation principles. Systems such as laser trackers that make use of time-of-flight distance meters in some cases are relatively more accurate than triangulation scanners, but non-contact triangulation scanners may be relatively faster because they typically project a plurality of light spots onto the object surface at each instant in time.
A typical triangulation scanner projects either a line of light (e.g., light from a laser line probe) or a two-dimensional (2D) pattern of light over an area (e.g., structured light) onto the object surface. In a triangulation scanner, a camera (e.g., a CCD or CMOS photosensitive array) is coupled to a projector (e.g., a laser light source) in a fixed mechanical relationship. The projected line or pattern of light emitted from the projector is reflected off of the object surface and imaged by the camera. Since the camera and projector are arranged in a fixed relationship to each other, the distance and angles to the object surface may be determined from the projected line or pattern, the captured camera images and a baseline distance separating the projector and the camera according to trigonometric principles. Triangulation systems provide advantages in quickly acquiring 3D coordinate data over large areas.
In some systems, during the scanning process, the triangulation scanner acquires a series of 3D images, which may be registered relative to each other so that the position and orientation of each 3D image relative to the other 3D images is known. If the scanner is stationary, such image registration is not necessary. Similarly, if the triangulation scanner is attached to, or works in conjunction with, a mechanical device having the ability to measure the position and orientation of the triangulation scanner, it is not necessary to provide such image registration. Examples of such mechanical devices include laser trackers, articulated arm coordinate measurement machines (CMMs), and Cartesian CMMs.
On the other hand, where the scanner is handheld and hence movable, various techniques may be used to register the images. One common technique uses features (e.g., cardinal points) located in the images to match overlapping areas of adjacent image frames. This technique works well when the object being measured has many features relative to the field of view of the scanner. However, if the object contains a relatively large flat or curved surface, the images may not properly register relative to each other.
Accordingly, while existing coordinate measurement devices are suitable for their intended purposes in working with triangulation scanners as described hereinabove, the need for improvement remains, particularly in improving the registration of images acquired by a triangulation scanner device.
Augmented reality (AR) is a relatively new type of technology that grew out of virtual reality. Augmented reality merges, superimposes, or transprojects actual real-world information or data with, on, into, or onto virtual information or data. That is, the virtual information or data “augments,” compliments or supplements the actual sensed, measured, captured or imaged real-world information or data related to some object or scene to give the user an enhanced view or perception of the real world object or scene. Augmented reality applications include technical or industrial areas such as part, component or device manufacturing and assembly and/or repair and maintenance, and facility, building or structure layout and construction. A number of modern-day AR applications are disclosed at http://en.wikipedia.org/wiki/Augmented_reality.
The actual information or data relating to the part, component or device or area may be obtained in various ways using various devices. One type of device includes a coordinate measurement device such as, for example, a CMM or a laser tracker. A camera may also be used to take still or video images of the actual part, component or device, and/or a desired area by itself or that surrounding or associated with the part, component or device.
The virtual information or data may be stored artificial information regarding the part, component or device. The stored virtual information or data may be related to the design of the part, component or device ranging from, for example, simple text or symbols to relatively more complex, graphic 3D CAD design data. Besides visual information, the stored virtual information or data may also comprise audible or sound information or data. The stored virtual information or data may also relate to information such as textual or part, component or device repair or maintenance instructions, or visual information depicting parts, components or devices that may be used, for example, in the design of an office or manufacturing and/or repair facility (e.g., a building or facility layout).
The combined actual and virtual information or data in an AR system is usually digital in nature and may be delivered in real-time (i.e., as the actual information is being measured or sensed) to a user on a display screen that may be in many different types or forms, such as that associated with, for example, a desktop or laptop computer monitor, tablet, smartphone or even a head-mounted display such as those associated with glasses, hats or helmets. Audio information may be delivered through a speaker.
While some innovations have already been made in the area of augmented reality for use with various types of devices, there is a need for novel applications of augmented reality together with handheld triangulation scanners (e.g., structured light scanners, laser line probes) used with a laser tracker.